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Abstract 



Latent space models (LSM) for network data were introduced by Hoff et al. (2002) 
under the basic assumption that each node of the network has an unknown position 
in a L'-dimensional Euchdean latent space: generally the smaller the distance between 
two nodes in the latent space, the greater their probability of being connected. In this 
paper we propose a variational Bayes approach to estimate the intractable posterior of 
the LSM. 

In many cases, different network views on the same set of nodes are available. It can 
therefore be useful to build a model able to jointly summarise the information given 
by all the network views. For this purpose, we introduce the latent space joint model 
(LSJM) that merges the information given by multiple network views assuming that 
the probability of a node being connected with other nodes in each network view is 
explained by a unique latent variable. This model is demonstrated on the analysis of 
two datasets: the excerpt of 50 girls from 'Teenage Friends and Lifestyle Study' data at 
three time points and the Saccharomyces cerevisiae genetic and physical protein-protein 
interactions. 

Keywords: latent space model, latent variable, multiplex networks, social network analysis, 
variational methods 
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1 Introduction 



Recently there has been a growing interest in the analysis of network data and statistical 



network models (see Salter- Townshend et al. (2012) for a recent review) have emerged as a 



major topic of interest in many research areas. Network data consists of a set of nodes and 
a list of edges between the nodes. 

Latent space models (LSM) are a well known family of latent variable models for network 



data introduced by Hoff et al. (2002) under the basic assumption that each node has an 
unknown position in a D-dimensional Euclidean latent space: generally the smaller the 
distance between two nodes in the latent space, the greater the probability of them being 
connected. 

Unfortunately the posterior distribution of the LSM cannot be computed analytically. 
For this reason we propose a variational Bayes inferential approach which proves to be less 



computationally intensive than the MCMC procedure proposed in Hoff et al. (2002) and can 
therefore easily handle large networks. 

In many cases, multiple network link relations on the same set of nodes are available. 



Multiple network views, also known as multiplex networks (Mucha et al. , 2010), can be 
intended either as multiple link relations among the nodes of the network or a single link 
relation observed over different conditions, such as one network evolving over time (longitu- 
dinal networks). 

In order to deal with multiplex networks we present a latent space joint model (LSJM) 
that merges the information given by the multiple network views by assuming that the 
probability of a node being connected with other nodes in each view is explained by a 
unique latent variable. 

To estimate this model we propose an EM algorithm: the parameter estimates obtained 
from fitting a LSM for each network view independently are used to find the joint posterior 
distribution of the LSJM; then these results are used to update the parameter estimates of 
every LSM. This process is iterated until convergence. 

This model has a wide range of potential applications. For example in computer science 



it is often of interest to jointly summarize the different relations (e.g. friend, fan, follower 



or like) that we can observe in social media like Facebook, Twitter and YouTube (Tang 



et al.l 2011). Another important application is systems biology where the joint modeling of 



physical and genetic protein-protein interactions is of wide interest ( Bandyopadhyay et al. 



2008). Other contexts in which this model can be useful include social sciences and business 



marketing (Ansari et al. , 2011). 

The LSJM is demonstrated on the analysis of an excerpt of 50 girls from 'Teenage Friends 
and Lifestyle Study' data at three time points, and two Saccharomyces cerevisiae networks. 

The paper is organized as follows. In Section [2] we introduce the data: in Section 2A_ 



we 



describe the excerpt of 50 girls from 'Teenage Friends and Lifestyle Study' and in Section 2.2 



we describe the Saccharomyces Cerevisiae Genetic and Physical Protein-Protein Interaction 
dataset. Section |3] provides an introduction to latent space models for network data with a 



particular focus on the variational Bayes inference approach to fit the model (Section |3.1 ) 
and in Section 3.2 we apply the LSM to the datasets. In Section |4] we introduce the latent 
space joint model. In Section [5] we illustrate the capabilities of the LSJM and we analyze 
its performance in the presence of missing values by using cross-validation. In Section |6] we 
conclude with a discussion of the model. 



2 Datasets 



2.1 Excerpt of 50 girls from 'Teenage Friends and Lifestyle Study' 



Pearson and Michell (2000) and Pearson and West (2003) collected data for a 'Teenage 



Friends and Lifestyle Study'. The dataset contains three directed networks about friendship 
relations between students in a school in Glasgow, Scotland. Each student was asked to 
name up to six best friends in the cohort. The data comes from three yearly waves, from 



1995 to 1997. An extended description of all the data in the study can be found in Pearson 



and Michell (2000). In this paper we will focus on an excerpt of 50 girls that were present 
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at all three measurement points. This dataset is available at the SIENA software websit^ 



(Ripley et al. (2011)) 



2.2 Saccharomyces Cerevisiae Protein-Protein Interactions 

In this paper we analyse a dataset containing two undirected networks formed by genetic 
and physical protein-protein interactions between 67 Saccharomyces cerevisiae proteins. The 
complex relational structure of this dataset has led to implementation of models aiming 



at describing the functional relationships between the observations (Bandyopadhyay et al. 



2008 [Troyanskaya et al. 2003). A list of proteins included in this dataset is displayed in 
Figure [T| The data were downloaded from the Biological General Repository for Interaction 
Datasets (BioGRID) database^ ( [Stark et al.] ( |2006| )). We refer to [Stark et al.| ( [2006[ [2011[ ) 
for a description of BioGRID, and for details regarding how the data were collected. 
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Figure 1: Saccharomyces cerevisiae proteins 
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3 Latent Space Model 



Latent space models for network data have been introduced by Hoff et al. (2002) under 
the basic assumption that each node n has an unknown position z„ in a D-dimensional 
Euchdean latent space. The distance model is an easy-to-interpret LSM which is based 
on the distance between the nodes in the latent space. Generally the smaller the distance 
between two nodes in the latent space, the greater the probability of them being connected. 
This model supposes the network to be intrinsically symmetric since it has the feature of 
being reciprocal: if Uij = 1 then the probability of yji = 1 is large, where i/ij is the observed 
variable that is = 1 if we observe a link from node i to node j, and = otherwise. For 
this reason, the distance model is particularly suitable for undirected networks or directed 
networks that exhibit strong reciprocity {yij = yji). The model can be written as: 

N N 

p(Y|Z, a) = Y[p{yij\zi, Zj, a) = YIy 

where we suppose that p{a) = 'N{C,,ip'^), p{zn) == N(0, cr^I/)) and cr^,^,-?/'^ are fixed parame- 
ters. The squared Euclidean distance measure is employed instead of the Euclidean distance 



exp(Q; — 








1 + exp(Q; 




Zj - 







used in Hoff et al. (2002). This choice has been made for two main reasons: firstly, it allows 
one to visualize more clearly the presence of potential clusters in the data, giving an higher 
probability of a link between two close nodes in the latent space and lower probabilities to 
two nodes lying far away from each other; secondly it requires less approximation to be made 
in the estimation procedure. 



The posterior probability is: 



N 



p = p{Z,a\Y) = p(Y|Z, a)p{a) llp{zn)C, 

71=1 

where C is a constant. 



3.1 Variational Bayes Approach 

Since the posterior distribution can not be calculated analytically we make use of a 
variational Bayes approach to estimate the model. To do this we aim at maximizing a 



lower bound of the likelihood function. This approach has been proposed for several latent 



variable models (Attias, 1999, Jordan et al. , 1999) and we refer to Beal (2003) for an extensive 



introduction to the variational methods. In the statistical network models context lAiroldil 



et al. (2008) proposed to use the variational method to fit mixed-membership stochastic 



blockmodels and Salter- Townshend and Murphy (2013) applied it to fit the Latent Position 



Cluster Model of Handcock et al. (2007): an extension of the LSM in which the latent 



positions are supposed to come from a Gaussian mixture model. 

We define the variational posterior q introducing the variational parameters B = ip"^) 
Zn and S: 

q = qiZ,a\Y) 



N 



n=l 



where q{a) = and g(z„) = >[(z„,S). 

The basic idea behind the variational approach is to find a lower bound of the log- 
likelihood by introducing the variational posterior distribution q and applying Jensen's in- 
equality: 



(Y) = logp(Y) = log j p(Y, Z, a) d{Z, a) 

p(Y,Z,a 



log / q 



d{Z,a) 



f , p(Y,Z,a) „^ , 
> / glog ^^ ' ' ^ d(Z,a) 

J q 

= logp(Y)-KL(g||p). 

The last line follows as p(Y) is neither a function of Z and a, and KL(g||j9) is the Kullback- 
Leibler divergence between the variational posterior q and the true posterior p. To estimate 
the model we aim to maximize the lower bound of the log-likelihood with respect to q: since 
only the second part of the lower bound depends on Z and a, this corresponds to minimizing 
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KL(g|b): 



KL(g|b) = - j q\og^d{Z,a) 

N 

= KL[q{a)Ma)] + ^ KL[g(z„)||p(z„)] - 



n=l 

2 J,2 £■^2 



2 I ^2 ^2 ^ ^2 



- log ^ + + ND hgia"") - Ariog(det(S)) 



where the expected log-hkehhood Eq[££] is approximated using the Jensen's inequahty: 
EJ££] =E,[log(p(Y|Z,a))] 

N 



= ^?/ijEja - \zi - Zjp] - Ejlog(l + exp(« - \zi - z^-p))] 

N 

^ - |zi - Zjp]) - log(l + Eg[exp(a - \zi - z^p)]) 



N 



J]l/,,(e-2tr(i:)-|z,-z,f) 



- log 



1 + ^ ^ exp ( -(z, - + 4S)-i(z, - ij] 



\ 



det(I + 4S)2 



To estimate the model an EM algorithm can be applied. The analytical form of the pa- 
rameter estimates will be found introducing the first and second order Taylor series expansion 
approximation of the following function: 



/(z„,S,e,Vi')=log 



1 + ^ ^-^ exp ( -(zi - + - ij] 



det(I + 4S) 



(2) 



calculated around the estimates calculated at the previous step of the algorithm (see Ap- 
pendix |A]) . 

Here we outline an EM algorithm on the {i + l)th iteration: 
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E-Step 



Estimate the parameters of the latent posterior distributions Zn^^'' and E*^ ^ evaluating: 

Q(e;e») = -KL(g|b) 
P 



glog- rf(Z,a), 



where 6 = (^j^/^^). This gives 



^(i+i) ^ N_ 
2 



2^ 



where J is the Jacobian matrix of f{zn\ S^*\ '?/'^ (Equation 2) evaluated at S = S*^*"*. 
And 



7(^+1) 



5^(y,. + y„,)z,-G(z«) + if( 



where G is the gradient and H is the Hessian matrix of f{zn\ S^'^^\ ip^^'^'') evaluated at 



Zji — Zn . 



M-Step 

Estimate the parameters of the posterior distribution of a evaluating 

e(*+^) = argmax 0(6; e«). 



This gives, 



where /' and /" are the first and the second derivatives of /(z„ , 
at f = f(^), and 



^(i)^^2(i)^ evaluated 



where /' is the first derivative of f(zn^'^\ J]^^~^^\ ^'^^^^) evaluated at ^'^ = ip^'^^'^. 
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3.2 Results 

3.2.1 Excerpt of 50 girls from 'Teenage Friends and Lifestyle Study' 

In this section we applied the LSM to the excerpt of 50 girls from the 'Teenage Friends 



and Lifestyle Study' data described in Section 2.1, The dataset consists oi K = 3 directed 



friendship networks on = 50 girls, collected at three time points. 

We fitted a LSM to each network separately assuming that p{a) = 3\f(0, 2) and p(z„) *= 
N(0,l2). The estimated latent posterior positions are shown in Figure |2] A matching 
rotation of the final estimates was applied in order to facilitate the interpretation of the 
results. The posterior distributions for the a parameters are quite similar over time q{ai) = 
K(-0.63,0.01), q{a2) = K(-0.66, 0.01) and ^(ag) = K(-0.48, 0.01); this means that there 
are not big changes in the density of the networks over time. From the analysis of the 
ROC curve (Figure [S] left) and AUG of the in-sample estimated link probabilities, it seems 
clear that the proposed LSM fits quite well the data separately. The boxplots in Figure |3] 
(right) show that the estimated probabilities distinguish quite well the true negatives with 
low probability of forming a link from the true positives with high probability of forming a 
link. 

3.2.2 Saccharomyces Cerevisiae Protein-Protein Interactions 

In this example we fitted the LSM to the Saccharomyces Cerevisiae data working with the 



genetic and physical interaction networks separately. The dataset described in Section 2^ 
contains K = 2 networks formed by genetic and physical interaction relations observed on 
A^ = 67 Saccharomyces cerevisiae proteins. We assume that p{a) = K(0, 2) and p{zn) = 
?sf(0,l2). The estimated latent positions are shown in Figure |4j The posterior distributions 
of the a's q{ai) = K(-0.332, 0.003) and q{a2) = K(-1.001, 0.005) indicate that network 1 
regarding the genetic interactions is much more dense than network 2 which refers to the 
physical interactions. The ROC curves, AUC and the boxplots (Figure |5| show that the 
proposed LSM fit the data quite well. 
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LSM Network 1 LSM Network 2 




n _ 



-3-2-10 1 2 3 

Figure 2: Latent positions z„ for the Girls Networks fitting tlie Latent Space Models. The 
color of the dots correspond to the proteins presented in Figure [1} The grey ellipses represent 
the 95% confidence intervals. Overlapping confidence intervals make darker shades of grey. 

4 Latent Space Joint Model 

Let us suppose to have K network views on the same set of nodes. We introduce 
a model which assumes that there is a continuous latent variable z„ ~ [N"(0, a^I^i) able 
to summarize the information given by all the network views Yi, . . . , Y^' identifying the 
position of node n in a D-dimensional latent space. In this case the network data are 
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Figure 3: ROC curves and boxplots of the estimated probabilities of a link for the true 
negatives and true positives obtained by fitting the Latent Space Models to the three Girls 
Networks 
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-3-2-10123 -3-2-1012 
Znl ^n1 



Figure 4: Latent positions z„ for the Genetic (left) and Physical (right) interaction data 
fitting the Latent Space Models. The grey ellipses represent the 95% confidence intervals. 
Overlapping confidence intervals make darker shades of grey. 

supposed to be conditionally independent given the latent variable. Our purpose is to model 
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Figure 5: ROC curves and boxplots of the estimated probabilities of a link for the true 
negatives and true positives obtained by fitting the Latent Space Models to the Genetic and 
Physical interaction data 



each network view by using a LSM. This yields to the following joint model: 

K 



p(Yi, . . . , Yx|Z,ai, . . . = ]^p(Yfc|Z;afe) 

k=l 
K N 

=nni 



exp(afc - 


Zi - 






1 + exp(Q;^ 




Zi - 







k=l ij^j 

where p{ak) = p(z„) *= N(0, cr^I^i) with a'^,C,k,'ipl set to be fixed parameters, and 

the dyad yij^ takes value 1 if there is a link between node i and node j in network k, and 
otherwise. 

The following identity allows one to find the model parameters ai, . . . , ax and the pos- 
terior distribution of the latent variable z„ given the K models: 



K 



p(z„| Yi, . . . , Yk] ax) oc 

fe=i 

oc p(z„) 11 



fe=i 



nf=iP(z«|Yfc;afc) 

p(Zn)^-l 



Applying the variational Bayes approach presented in Section [3TT] introducing the variational 
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parameters ©fe = (Cfe^V'fe)' ^nk ^nd S^, it is possible to approximate p{zn\y k] o^k) with 
g(z„) ~ 3\f(z„fe, Sfc). And since z„ ~ 3\r(0, o-^Id), we obtain that the posterior distribution of 
the latent variables given all the network views can be written as: 

p(z„|Yi, . . . , Yk; 6i, . . ■ , Qk) oc X(z„, S), 



where the parameters are 



and 



K 



K -1. 



z„ = S 



fe=i 



K 



-1 



K -1. 



IdO 



.k=l 
■ K 

^Sj. Z„fc 
.fe=l 

By fitting LSJM we get information on p(z„|Yi, . . . , Y^^-; ©i, . . . , Ok) and p(z„|Yfe; ©fe) for 
k = 1, . . . , K. This way it is possible to have estimates for both the overall positions z„ and 
the position given one particular network view z^k- 

The estimates of Znk and Sjt are updated from z„ and S, so we can locate the uncon- 
nected nodes or subgraphs in the latent space depending on their position conditional on the 
other network views, avoiding the usual tendency of pushing away the unconnected nodes 
to maximize the likelihood when fitting the classical LSM. This approach also allows one to 
plot the positions given each network view in the same latent space, and to look at how the 
nodes in each network change the positions. 

Let \ . . . , ©x ) current estimates of (©i, . . . , ©x) and initialize (©i'''*, • • • , ©^^)- 

The EM algorithm at the iteration {i + 1) can be summarized as follows: 
E-Step 



Estimate the parameters S^'"^^^ and z^^^^ of the posterior distribution of the latent 



13 



variables given all the network views evaluating: 



Q(ei,...,e^;e«,...,e 



K , 



p(z|Y„...,Y,,;e«,...,e«)[log(^(Yi, . . . , Y;^, Z|ei, . . . , 6,^))] 



K 
k=l 



p(Z|Y„...,YK;0«,...,e«)[log(^'(Yfc.Z|0fc))] 



(3) 



-(^-l)W....,YK;e«,...,e«)[log(P(Z))]- 
Thus we can estimate the parameters of the posterior distribution p(z„|Yfc;Ofc) given 
each network separately: 



2-1 u — — 



2^ 



N 



i=l jyti 



where Jk is the Jacobian matrix of f{zn\ s''*\ ^fc*'*'') (Equation 2) evaluated at S = S*'*^ 
and. 



2a2 





-1 - 







Y^iVjuk + ynjk)zf - G,(Z«) + i/.(z»)z« 



where Gk and Hk are respectively the gradient and the Hessian matrices of f{zn \ ^l!\4>l^^'') 
evaluated at z„ = Zn'' ■ 

The posterior distribution of the latent positions given all the network views is estimated 
merging the estimates of the single models: 

p(z„|Yi, . . . , Yk] 01, ... , Qk) oc K(z„, S), 



where 



and 



■ K 

El 

.k=l 



K -1. 



■ K 

El 

.k=l 



.^k J ^nk 
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M-Step 

Update the model parameters evaluating 



(G^^\ . . . , 9^+^)) = argmax 0(81, . . . , 9^; e«, . . . , e«). 

This gives 

_ ^A: + V'fc(X]i=l ^j^i Vijk — f'ki^k ) + fki^k )) 

where and are the first and the second derivatives of f{zn^^\ S*'*^^'', ^^^^ ■i/'^*'*'') evaluated 
at f = and 

-1 



where is the first derivative of f{zn~^^\ ^p^^^^) evaluated at ip"^ = ipl^^"^- 



5 Applications 

5.1 Computational Aspects 

The LSM and the LSJM have been fitted assuming that p{a) = 3\f(0, 2) and p(z„) *= 
?\f(0,l2), initializing the variational parameters ^fc = and ipl = 2, and latent positions z„ 
by random generated numbers from 3\f(0,l2) and setting S = I2. 

Ten random starts of the algorithm were used and the solution with the maximum like- 
lihood value was selected. 

The latent positions are identifiable up to a rotation of the latent space. For this reason 
to speed up the convergence of the algorithm we matched the estimates of Znk V/e > 1 with 
Zni via a Procrustes transformation in the first 10 iterations of the EM algorithm. 

The EM algorithm was stopped after at least 10 iterations when 

\\Eg[eef+^^ -Eq[u]^^\\ < toi, 

where Eg[££] is given by Equation [T] if we fit the LSM or by Equation [s] if we fit the LSJM, i 
indicates the iteration, tol is a desired tolerance value (which, in this case, was set tol = 10^^). 
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To assess the fit of the model we evaluated the in-sample predictions producing the ROC 

curve of the estimated link probabilities and calculating the area under the curve (AUG) 

and the boxplots of the estimated link probabilities for both the true positive and the true 

negative links. We estimated the probability of a link under each network view by calculating: 

/ , |. . r . exp(4 - |zifc - zjfcp) 

PKVijk = l|Zjifc,Zjfe,^feJ = 7~ 

1 + exp(^fe - \zik - Zjk\^) 

5.2 Missing Data 

Missing hnks can be easily managed by the LSJM using the information given by all the 
network views. To estimate the probability of a missing link we employ the posterior mean 
of the ak and of the latent positions so that we get the following equation: 

V* -v(v.-l\z z D- ^Mik - \Zi - 

1 + exp(^fe - \Zi - Zj\^) 

If we want to infer whether to assign yijk = 1 or not, we need to introduce a threshold Tfc, 
and let Uijk = 1 if p{yijk = l|zjfe, Zjk, ^k) > ^k- We set the threshold to be equal to the median 
probability of a link for the subset of the actual observed links in network k. 

To evaluate link prediction in the presence of missing hnks, we used a 10-fold cross 
validation procedure consisting of randomly splitting the set of all the possible dyads in each 
network view into 10 subsets. Then we can predict the links of each subset given the others 
fitting a LSM to each network independently and then fitting the LSJM. We compare the 
link prediction performance given by these two methods. 

The LSJM allows one to locate in the latent space a missing node (no information about 
links sent and received by the node) , in one network by employing the information provided 
by the other network views. We evaluate the link prediction for the missing nodes applying 
a 10-fold cross validation on the nodes, randomly dividing the set of nodes in each network 
into 10 subsets, and then predicting the links using the LSJM. We do not use a single LSM 
to locate missing nodes in the latent space and to estimates their probabilities of links since 
the only information that the model would use for the estimates of their latent positions 
would be the prior distribution of the nodes p(z„). 
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To facilitate the interpretation of the results we matched the rotation of the latent posi- 
tions in the single LSM with the ones obtained from the LSJM. 



5.3 Excerpt of 50 girls from 'Teenage Friends and Lifestyle Study' 



In this application we fitted the LSJM to the Girls dataset presented in Section 2.1 The 
overall positions z„ in the latent space are displayed in Figure [6] In Figure [6] (right) the dots 
represent the overall positions z„ and the arrows connect the estimated positions z„fc under 
each model k from network A; = 1 to /c = 3, so that it is possible to see their variation over 
time. 



LSJM 



• LSJM: Net1 -* Net2 -* Net3 





Figure 6: On the left are shown the latent positions z„ for the Girls Networks fitting the 
LSJM, the grey ellipses represent the 95% confidence intervals. Overlapping confidence 
intervals make darker shades of grey. In the plot on the right the dots represent the overall 
positions z„ and the arrows connect the estimated position under each model z„i,z„2,z„3. 



Figure [7] shows the estimated positions z„fc under each model in the latent space. These 
plots allow a direct comparison between the positions in the latent space Znk given by 
the LSJM and the ones obtained by fitting a single LSM for each network view (Fig- 
ure [2]). It is interesting to observe that in the LSJM context the proximity between the 
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disconnected components in network 3 depends on their previous relations. The posterior 
probabihties of the model parameters q{ai) = IN"(— 0.42, 0.01), q{a2) = !N(— 0.39, 0.01) and 
^(as) = !N(— 0.32, 0.01) are lower than the single LSM approaches (Section |3.2.1 ), implying 
that a given distance d between two nodes in the latent space corresponds to a higher prob- 
ability of a link. From the ROC curves, AUG and the boxplots (Figure pi) of the estimated 

LSM Network 1 LSM Network 2 





LSM Network 3 




Figure 7: Estimated latent positions under each model z„i,z„2,z„3 for the Girls Networks 
fitting the LSJM. The grey ellipses represent the 95% confidence intervals. Overlapping 
confidence intervals make darker shades of grey. 



probabilities of a link, it appears clear that the LSJM fits the data quite well. To evaluate 
the link prediction we applied a 10-fold cross validation setting 10% of the links to be missing 
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Figure 8: ROC curves and boxplots of the estimated probabilities of a link for the true 
negatives and true positives obtained by fitting the LSJM to the Girls Networks 



at each time point. The ROC curves and AUC in Figure |9] show that the estimates of the 
links are quite good, especially when applying the LSJM. 



ROC curve Overall 



ROC curve Overall 























;;■ 










i( 
r' 










■( 

!' 
■( 

- H 

! 

- i 
I 




true positive r 

0,2 0.4 1 


■? 
l> 
i 

! 
1 




1 


- - AUC network 1 = 94 % 

AUC networit 2 = 91 % 

■ - ■ AUC nelworl< 3 = 94 % 




! 
f 


- - AUG network 1 = 86 % 

AUG network 2 = 91 % 

■ - ■ AUG network 3 = 94 % 



false positive rate 



false positive rate 



Figure 9: Girls Networks: ROC curves for the 10-fold cross validation for missing links fitting 
a LSM (left), and 3 single LSM (right) 



Setting p{yijk = ^\i'ik, ijk,^k) > Tk where is equal to the median probability of a link for 
the subgroup of the actual observed links in network k and applying the LSJM we obtained a 
misclassification rate of 4% for every network, whereas applying three single LSM we obtain 
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a misclassification rate of 4% for network 1, and 5% for network 2 and 3. 

The LSJM allows one to also manage missing nodes, to do this we applied a 10-fold 
cross validation setting the 10% of the nodes in each network to be missing. We obtained 
a misclassification rate of 9% for all the three networks. As mentioned above in this case 
the LSM approach would be useless since it would locate the nodes only relying on the prior 
information. 



5.4 Saccharomyces Cerevisiae Protein-Protein Interactions 

In this section we applied the LSJM to the Saccharomyces Cerevisiae dataset (see Sec- 



tion 



2.2). Figure 10 shows the estimated overall latent positions z„ and in the plot on the 



right each arrow starts from the positions z„i for the genetic interaction dataset and points 



to the latent positions z„2 for the physical interaction dataset. Figure 11 shows the estimated 



LSJM 



• LSJM: Genetic Physical 



















• o 









Figure 10: On the left are shown the latent positions z„ for the Saccharomyces Cerevisiae 
Networks fitting the LSJM, the grey ellipses represent the 95% confidence intervals. Over- 
lapping confidence intervals make darker shades of grey. In the plot on the right the dots 
represent the overall positions z„ and the arrows connect the estimated position under each 
model z„i,z„2- 
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position under each model z„fc in the latent space. It is possible to compare these results 
with Figure |4] in which we have fitted two single LSM to the data. The ROC curve, AUG 



LSM Genetic Interactions LSM Phiysical Interactions 




-3-2-10123 -3-2-1012 



Figure 11: Estimated latent positions under each model z„i, z„2for the Saccharomyces Cere- 
visiae Networks fitting the LSJM 

and the boxplots in Figure [12] show that the LSJM fits the data quite well. The posterior 
distributions for the a's are q{ai) = K(-0.410, 0.003) ^(as) = X(-0.940, 0.005). 

We applied a 10-fold cross validation to evaluate the prediction of missing links. In 
Figure [T3] are shown the ROC curve and the AUG obtained fitting the LSJM and two single 
LSM respectively. The results of the LSJM show a much better fit in terms of estimates 
of missing links for the genetic interaction network compared to the ones obtained by the 
single LSM approach. Setting the threshold to be equal to the median probability of a link 
for the subgroup of the actual observed links we obtained similar misclassification rates for 
the missing links in both LSJM and single LSM approach: 9% for the genetic interaction 
network, and 6% for the physical interaction network. 

Applying a 10-fold cross validation for missing nodes using the LSJM we obtained a 
misclassification rate of 24% for the genetic interactions dataset and 20% for the physical 
interaction network. 
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ROC curve 



Genetic Interactions 



Physical Interactions 



AUG Genetic Interactions = 94 % 
AUC Pliysical Interactions = 93 % 



false positive rate 



Figure 12: ROC curves of the estimated probabilities of a link for the true negatives and 
true positives obtained by fitting the LSJM to the Saccharomyces Cerevisiae Networks 
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AUC Genetic Interactions = 90 % 
AUC Physical Interactions = 88 % 
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Figure 13: Saccharomyces Cerevisiae Networks: ROC curves for the 10-fold cross vahdation 
for missing hnks, fitting the LSJM (left) and fitting 2 single LSM (right) 
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6 Conclusions 



The large amount of network data requires the introduction of new models able to describe 
their complex connectivity structure. On the other hand efficient inferential methods are 
needed to carry out estimation efficiently. 

In this paper, we proposed a latent variable model (LSJM) for multiple network views 



that extends the latent space model proposed by Hoff et al. (2002) allowing to summarize in 
the same latent space the information given by different relations on the same nodes. The 
use of the variational approach for the computation of the model parameters allows us to 
apply the latent space joint model to larger networks (of the order of thousands of nodes). 

This model allows to find the position of each observation in a latent space based on all 
the available information in the datasets. Further information like the latent positions in 
each network view separately are obtained updating the single- network estimates given the 
overall positions. These information allows effective visualization and prediction of the data. 

The examples presented show how the LSJM facilitates the interpretation of the different 
positions of the network nodes in the latent space according to longitudinal measurements 
in the excerpt of 50 girls from 'Teenage Friends and Lifestyle Study' example and multiple 
relations in the Saccharomyces Cerevisiae genetic and physical protein-protein interactions 
dataset. 

Future work may lead to an extension of the model allowing cluster formation by assuming 
that the latent positions come from a Gaussian mixture model fitting each network using 



a Latent Position Cluster Model (Handcock et al. , 2007). Or to build a model that takes 



explicitly into account the sequential feature like dynamic network analysis does (Sarkar and 



Moore 


2005 


Hoff 


2011 
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A Estimate of the Parameters LSM 



A.l E-Step LSM 



1 Estimate ir. 



KL < Consts„ + z^z„ , ^ „ 



+ 2^1og 



( exp K + 
1 + — exp [-{in- Zi)^(I + 4S)-^(z„ - % 



V 



det(I + 4S)2 



Second-order Taylor series expansion approximation of 



Therefore, 



1 + — ^ -^J- exp ( -(z„ - z,/(I + 4S)-^(z„ - z,) ) 



v 



det(I + 4S)2 



/ 
\ 



/(Zn) ~ /(Zn) + (Zn " Z„)^G(z„) + ^ (z„ - Z„)^if (z„) (z„ - Z^ 



Let find the gradient G and the Hessian matrix H oi f evaluated at z„ = z„. 



G(z„) = -2(I+4S)-i^(z„-z,) 



1 + "^"'f^^f^' exp ((z„ - z,)^(I + 4S)-^(z„ - z,)) 



exp ( C + 2 V'^ 



-1 



i/(z„) = -2(I + 4S)-i J] 



1 + ^'^^^^^f^^l exp ((z„ - zjfil + 4S)-^(z„ - g,: 



exp ( C + 



2(z,-z,-)(z„-%f (I + 4S)-i 



exp I (+--il>^ 
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Therefore, 



aKL ^ 



+ ^iVjn + Vnj) ] I + H{Zn) 



2a2 



^{yjn + ynj)Zj - G{Zn) + -H"(z„)z„ 



+Conf 



2(72 



2 Estimate S 



J¥=n 



N 



KL < Const^ + tr(S) ( A + 2 ^ J]] y,, ) - ^ log(det(S)) 



AT 



^ exp (1 + 
1 + — ^ ^ exp ( -(zi - z,)^(I + 4S)-^(zi 



trS^ det(I + 4S). 

First-order Taylor series expansion approximation of: 



N 



( exp K + 
1 + — ^ ^-f exp ( -(zi - z,)^(I + 4So)-'(zi 



V 



det(I + 4So)2 



/(S)«^/(So) + J(So)(S-So) 
where J is the Jacobian matrix of / evaluated at S = Sq. 



J(So) = 4(1 + 4So)-^ E E ((^^ - %)(^^ - + 4So)-^ - ^l) ■ 



det(I + 4Eo 



exp ( C + 2 



1 + y 77"\ exp ((z, - z,)^(I + 4So)-^(z, - z,) 
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Therefore, 



KL ^ tr(S) I ^ + '^J2Y1 - Y log(det(S)) + J(So)S + Constf, 



A.2 M-Step LSM 

1 Estimate ^ 



- 1^ " M 1^ + 2] E + 5] 5Z + exp(0/l,,) + Const^ 



where = exp Q^^^ det(I + 4S)~5 exp (^-(zj - Zj)'^{l + 4S)-^(zj - z^)^ . 



Second-order Taylor series expansion of: 



i=l j^i 

evaluated at ^ = 



N 

fi^o) = J] J] log (l + exp(lo) A,,) 



/(O « /do) + /'(|o)(e - 6) + (eo)(e - 6)' 



^ -1 



where 

N ^ _i ^ _1 

/"do) = E E (l + ^M-io)Ar^') (l + expdo) A,,) 

Therefore, 
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2 Estimate t/)^ 



KL < 



1 



2V^2 2 



log(Vi') + ^ ^ log + exp ('^^''l A,^ + Const^2 



where = exp(^) det(I + 4S)-^ exp (^-(zj - + 4S)-^(zj - z^)) 



First-order Taylor series expansion of: 

N 



f(^") = E E (i + (^^o) 
1=1 V \ / / 



evaluated at -0 = V'o 



where 



Therefore, 



i=i j^i v \ / / 



dKL 



= 



KL ^ + fi^l)) - I log(^') + Const^. 



-1 
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